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"Method for compressing/decompressing video information" 

FIELD OF THE INVENTION 

The present invention relates to a method for compressing/decompressing video 
information, and to corresponding compressing and decompressing devices. It also relates to 
a computer program product for carrying out said method, to compressed data obtained by 
5 implementation of said method, and to devices for compressing and decompressing video 
information. 

BACKGROUND OF THE INVENTION 

The current standards belong to the MPEG family like MPEG-1, MPEG-2 and 
MPEG-4 (see for mstance MPEG-4 Visual codmg standard ISO/IEC 14496-2, document 

10 available at ISO, referred here imder the MPEG-4 document number w3056) and to the ITU 
H.26X family like H.261, H.263 and extensions, and H.264. 

Most video coding standards take advantage of the fact that there are some 
redundancies between subsequent frames of a video sequence. In most video compression 
algorithms, each frame is subdivided into segments which can be regular square blocks, as in 

15 MPEG-4, or square or rectangular blocks as in H.264. When decompresshig the compressed 
data, each segment of a following frame is obtained by calculation of a prediction, based on a 
corresponding segment of a previous frame by using motion information generally called 
motion vector, and correction or residual information generally called residual image 
defining the differences between the segment and its prediction and more generally between 

20 the frame and its prediction. The compression standards provide a way of encoding the 

motion information and the correction information for retrieving a following frame based on 
a previous known frame. 

Mainly, two approaches are used by the compression standards. A first approach is 
named the backward approach. It is implemented in MPEG and ITU H.26X standards. In the 

25 backward approach, for each segment of a following frame, the compression method tries to 
find the segment of the previous frame that is the closest to it or at least less far from it. This 
may cause problems for segments that appear in the following frame and not in the previous 
frame. Still, for any segment that is present in both the previous and the following frames, 
this system gives the best prediction, for a given motion model, providing a full search of the 

30 motion parameters is done, that is to say that any possible parameters are considered. The 
problem with the backward approach is that, in many cases, the used frame segmentation 
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does not coincide with the frame segmentation into "real" objects that may have different 
overall motions. 

The second approach is named the forward approach. This approach is disclosed for 
example in WO-00/64167. It uses a segmentation based coding scheme using "real" objects 
5 of the frame. In the forward approach, for each segment of a previous frame, which is 

considered as an independent object, the best match is searched in the following frame, that is 
to say the method tries to find what the object became between the two frames. The segments 
of the following frames that are predicted are not optimally predicted with regard to the 
considered motion model. It is not sure to have the best possible prediction of the following 
1 0 frame from the previous frame, even with frill search motion estimation. Indeed the 

optimisation is performed regarding what one has, i.e. the previous frames, and not regarding 
what one wants to have, i.e. the new frame. 



SUMMARY OF THE INVENTION 

15 It is therefore the object of the invention to provide a video compression metitiod - 

and an associated decompression method - that enables to optimise the segmentation used for 
the prediction of a following frame based on a known previous frame. 

To this end, the invention relates to a method for compressing video information in a 
video sequence (It, It+i) comprising the steps of : 
20 - considering in said sequence a first video frame (Bt) containing image data ; 

. segmenting said first video frame (Bt) into segments (St,i) ; 
. for each segment (St,i) of the first video frame (Bt) : 

- searching, in a second video frame (It+i) following the first video frame (Bt) in 

the video sequence, a corresponding predicted segment (SF44°J^^^^ ) which matches with the 
25 segment (St,i) of the first video frame (Bt) according to a predetermined similarity measure ; 

- calculating a raw set of motion parameters (Mt,i ) describing the motion 
between the segment (St,i) of the first video frame (Bt) and the corresponding predicted 

« 

segment ( SF+^^r'^^^ ) of said second video frame (It+i) ; and 

. for each corresponding predicted segment (S?4.^^i^^^^ ) of the second video frame 

30 (Itfi): 
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- searching, in the first video frame (Bt), a corresponding segment (sj>.i3ackward) 

that matches with the predicted segment ( S&jj'^^'*^ ) of the second video frame (It+i) according 
to a predetermined similarity measure ; 

- calculating a best set of motion parameters (Mt,i ^M?,i ) describing the 

5 motion between the corresponding segment ( sf ) of the first video frame (Bt) and the 

predicted segment (Sft.i°r^'"*^ ) of the second video frame (It+i), said best set of motion 
parameters consisting in the raw set of motion parameters (M ^ ; ) corrected by a motion 

parameters correction ( AM^ | ). 

As seen later in the description, such a method has the advantage to provide a 
10 segmentation of the following frame which is optimised before applying a backward 
approach to determine the best set of motion parameters. 

Additional features are recited in the sub-claims 2 to 8. 

Another object of the invention is to propose a method for decompressing video 
information in a video sequence (It, It+i) comprising : 
1 5 . considering a first video frame (Bt) containing image data; 

. segmenting said first video frame (Bt) into segments (St,i); 

. for each segment (St,i) of the first video frame (Bt), defining a projected segment 

( St+i.i ) t>y applying to the segment (St,i) of the frrst video frame (Bt), a raw set of motion 
parameters ( M^i ) describing the motion between the segment (St,i) of the first video frame 

20 (Bt) and the corresponding projected segment ( Sf+u ) and 

. for each corresponding projected segment (St+i,i)* 
- finding in the first video frame (Bt) a corresponding improved segment ( Su ) using both the 
raw set of motion parameters (Mt,i ) ^d a motion parameters correction (AM^ j ), the 
corresponding improved segment (s^ ) being the segment of the first video frame (Bt) that 

25 would be projected on the corresponding projected segment ( Sf+u ) by applying to it the raw 
set of motion parameters (Mli ) corrected by the motion parameters correction (AMfj) ; and 
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- defining a corrected projected segment (St+u ) applying the raw set of motion 
parameters ( M?,i ) corrected by the motion parameters correction (AM ^ j ) to the 

corresponding improved segment (s^ )• 

It also concerns a computer program product for a data processing unit, comprising a 
5 set of instructions, which, when loaded into said data processing unit, causes the data 
processing unit to carry out the compressmg method previously mentioned. 

In addition, the invention also relates to a device for compressing video information 
in a video sequence (It, It+i) comprising : 

- means for segmenting the first video firame (Bt) containing image data into 
10 segments (Sy); 

- means for searching, in a second video frame (It+i) following the first video firame 
(Bt) in the video sequence, a corresponding predicted segment (S?4°r^'^'^ ) which matches with 

the segment (St,i) of the first video frame (Bt) according to a predetermined similarity 
measure, for each segment (St,i) of the first video firame (Bt) ; 
15 - means for calculating a raw set of motion parameters ( M\\ ) describing the motion 

between the segment (St,i) of the first video frame (Bt) and the corresponding predicted 

segment (S?4.t,r^'''^ ) of the second video firame (It+i), for each segment (St,i) of the first video 

frame (Bt) ; 

- means for searching, in the first video frame (Bt), a corresponding segment 

20 (sf jbackward ) that matchcs with the predicted segment ( Sf+jJ'^^''^ ) of the second video frame 
(It+i) according to a predetermined similarity measure, for each corresponding predicted 
segment (S?44°i^^^*^ ) of the second video frame (It+i) ; 

- means for calculating a best set of motion parameters (Mt,i ^M?,i ) describing 

tlie motion between the corresponding segment ( sfj^^^^^^^ ) of the Gxst video frame (Bt) and 

25 the predicted segment ( Sf^i^r^'"* ) of the second video frame (It+i), said best set of motion 
parameters consisting in the raw set of motion parameters (Mf j ) corrected by a motion 

parameter correction ( AM^j), for each corresponding predicted segment ( Sf^i^r^"*^ ) of the 

second video frame (It+i). The invention still relates to a device for decompressing video 

information in a video sequence (It, It+i) comprising : 
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- means for segmenting said first video jtame (Bt) containing image data into 
segments (St,i); 

- means for defining a projected segment ( St+i.i ) ^^^^ segment (St,i) of the first 
video frame (Bt), by applying to the segment (Stj) of the first video firame (Bt), a raw set of 
motion parameters ( M^i ) describing the motion between the segment (St,i) of the first video 

frame (Bt) and the corresponding projected segment (St+u ) J 

- means for finding, in the first video firame (Bt), a corresponding improved segment 
(Su ) using both the raw set of motion parameters (Mt,i ) a motion parameters correction 

(AM t,i ), the correspondmg improved segment (Sy ) being the segment of Bt that would be 
projected on the corresponding projected segment (Sf+ij ) by applying to it the raw set of 
motion parameters (M^i ) corrected by the motion parameters correction (AMJ^, ), for each 
corresponding projected segment ( St+i,i ) J and 

- means for defining a corrected projected segment (St+^^i ) by applying the raw set 
of motion parameters ( Mf,i ) corrected by the motion parameters correction (AM ) to the 

corresponding improved segment (Su )> ^ot each corresponding projected segment (St+i,i )•• 

It also concerns compressed data corresponding to a video sequence as obtained by the 
previously mentioned compressing method. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be now described, by way of example, with reference to 
the accompanying drawings in which : 

- Figure 1 is a schematic view of a typical processing chain for a video sequence 
made of successive frames ; 

- Figure 2 is a flow chart showing the video compression algorithm of the method 
for encoding a sequence of frames according to the invention ; 

- Figure 3 is a schematic view of a processed firame at successive steps during the 
compression operation ; 

- Figure 4 is a flow chart showing the video decompression algorithm of the method 
for decoding the compressed data corresponding to the compression method of a sequence of 
frames according to the invention ; and 
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- Figure 5 is a schematic view of a processed frame at successive steps during the 
decompression operation. 

DETAILED DESCRIPTION OF THE INVENTION 

Figure 1 shows a typical processing chain for a video sequence made of successive 
frames, each frame containing image data. This chain includes an encoder 12 adapted to 
receive a sequence of frames. These frames, provided for example by a digital camera, are 
arrays of pixels, each pixel being characterized by color parameters, which can be 
chrominance and luminance or Red, Green and Blue values for example. In the following, the 
t^^ frame of an input sequence is denoted It. 

The encoder 12 is adapted to implement a video compression method according to 
the invention and to output encoded data that are compressed. The compression method takes 
advantage of the fact that there are some redundancies between subsequent frames of the 
video sequence. The encoded data are then stored in a support like a tape or are transmitted 
over a medium like a wireless network. The processmg chain finally comprises a decoder 14 
that is adapted to decompress the encoded data and to provide a sequence of frames. The 
decoder 14 is adapted to implement a video decompression method according to the 
invention. In the following, the t*^^ frame of a decompressed sequence is denoted Bt. 

Figure 2 shows the video compression algorithm carried out in the encoder 12. 
Figure 3 shows the following processed frames during the implementation of the 
compression algorithm. This algorithm of Figure 2, implemented by a processing unit like a 
DSP driven by an adapted software program, is repeated for each frame of the sequence of 
frames to be encoded, The steps shown in Figure 2 concern the encoding of frame It+i. It is 
assumed that the previous frame It has aheady been encoded and that the decompressed 
frame Bt corresponding to frame It is known. 

Roughly, the method consists in two main stages. In a first stage 200, a first set of 
motion parameters is defined by usmg a forward approach to project segments of the 
previous frame It in order to predict the following frame It+i. . Once projected, a predicted 
segmentation of the following frame It+i is provided. This segmentation is more likely to 
coincide with real objects of the following frame It+i than an arbitrary segmentation, a grid of 
blocks for example. Without considering the "holes" of the predicted segmentation as 
segments (they are not predicted by any part of the frame It), for each projected segment, a 
corresponding segment called a predicted segment is provided in the prediction, and above 
all, a set of motion parameters. In order to refine the prediction if necessary, a new motion 



wo 2005/079074 



PCT/IB2005/000049 



estimation is performed in a second stage 201, this time by using the backward approach on 
the predicted segments. 

The first stage 200 is based on a forward approach, which means that segments of 
the previous frame It are searched in the following frame It+i. The compression method will 
5 now be disclosed in details. 

At step 202, a segmentation of decompressed frame Bt is defined and stored. The 
segmentation of Bt consists in defining a subdivision of Bt into segments St,i. A set of 
segmentation parameters is determined. It defines the segmentation process that is 
implemented. Advantageously, the segment boimdaries coincide with object boundaries in 
10 the frame Bt. Thus, the segments are "real" objects in the picture corresponding to frame Bt. 
In the following, all the segments St,i are processed subsequently. Thus, a segment St,i is 
considered in Bt at step 204. 

At step 206, a corresponding segment S t+li is searched in the following frame It+i to 

be encoded. The corresponding segment Stfi,i is the segment of frame It+i which provides 

15 the best match with segment St,i according to a given similarity measure. The similarity 
measures are known per se and will not be disclosed. 

At step 208, some parameters that enable to retrieve S m,i ^® stored. In particular a 
raw set of motion parameters M^i is calculated. These motion parameters define the changes 
in the position of S t+i,i with respect to St,i. For example, the set of motion parameters 
20 Mil defines a translational motion. According to the forward approach, the predicted segment 
denoted Sf;^°7^''^ and the raw set of motion parameters Mf j are defined as follows : 

SF+t.r^'''^ = MC(St,i, M^j ), where MC(Stj, M^j ) is the motion compensation 
operation using j as motion parameters ; 



where S t4-i,i,M is segment of the following frame It+i that coincides with MC(S {^^ , M) and 

diff(a,b) measures the similarity of a and b (the higher the measure, the less a and b are alike 
— it can be, for example, the sum of square differences of the pixel color values over the 
30 whole segment) ; 




arg min (diff (MC(St,i, M),Sf/i,i,M)) 



(for example) ; 



25 



M in motion 
search range 
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SF-i-u'^^'^^ is a raw predicted segment. 

At this step, no processing is done to deal with overlapping between the different 
predicted segments gfi^^i^^''^ . The overlapping issues are dealt with as disclosed below. Steps 
202-208 are repeated until all the segments St,i of previous decompressed frame Bt are 
5 considered. Thus, for each segment St,i, a corresponding predicted segment Sfc-f^^f^^*^ is 

defined in the prediction together with a raw set of motion parameters M^i called forward 
motion parameters. 

Once every segment of the previous decompressed frame Bt is projected, some raw 
predicted segments may overlap each other. Some decisions is made at step 210 for solving 

10 the intersection between adjacent segments. Thus, overlapping parameters are calculated. 

According to a first embodiment, it is determmed which segment is in front of which. How to 
make these decisions is not within the scope of the present invention. When it is done, 
predicted segments have their final shape, i.e. the original shape minus the possibly hidden 
parts. According to another embodiment, a merge parameter a is determined for each pair of 

15 adjacent segments. For each pixel of the intersection between the adjacent segments, a value 

of the pixel Poverlap is defined by : Poverlap = aPsegmentl + (1 - a) Psegment2> whcrC Psegmentl 

and Psegment2 are the values of the corresponding overlapping pixels of both segments. 
The prediction of the following frame Ihi resulting from the set of predicted 

segments Sf44°i^^^^ niay have holes. These holes correspond to newly uncovered parts of the 

20 following frame It+i. The holes are considered as new predicted segments or their contents 

are stored at step 212. Holes processing is not within the scope of the invention. In a possible 
embodiment, holes are added to the projected segmentation as new segments to be processed 
in the next step of the algorithm. They could also be merged with an existing predicted 
segment, or simply be stored as holes to be processed after motion processing was done. In 

25 any case, infonnation regarding holes is encoded and stored. After step 212, a set of Sf+fr^**^'^*^ 

segments that corresponds to the set of SF44°,i^^^^ segments after having been processed for 

holes and overlapping is defined. At step 213, it is decided whether a calculation of a new 
best set of motion parameters by using a backward approach is necessary. If it is necessary, 
the second stage 201 is performed and a corresponding flag [YES] is memorized. If not, a 
30 corresponding flag [NO] is memorized and step 220 is directly performed for calculating 
residual frame Rt+i as will be disclosed below. 
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If necessary, a new motion estimation is perfomied in the second stage 201, by using 
a backward approach based on the predicted segments gp^^pdicted step 214, a predicted 

segment sp^^pdicted provided by the forward approach is considered. At step 216, a search is 
carried out in the previous jframe Bt to find the segment of frame Bt denoted Sf j ^^''^'^^''^ that is 
5 the closest to the considered predicted segment S^i^^*"*^'^^ according to the given similarity 
measure. At step 218, the new best set of motion parameters denoted MF,i + ^M?,! is 
calculated. A M^i is a motion parameters correction such that M?,i + A M^i defines the motion 

from sf,!^^''^^^^ to Sf+ff^^""^^ . The new best prediction for sP|^^*^^^^^ is searched in a small area 
around the segment defined by applying the forward motion parameters M^i to the predicted 

10 segment gp^fjedicted steps 214-218 are repeated until all the predicted segments Sf+fJ^'^'''*®'^ are 

considered. At this point, remaining holes are processed. 

At step 220, a residual frame Rt+i is computed and encoded. The encoding method is 
not within the scope of this invention. In a possible embodiment, residual frame Rt+i may be 

encoded on a segment basis using the projected segmentation (Sf+f^'^'''^^'^ ^.nd holes). The 
15 residual frame Rt+i defines the structural differences between the image prediction BPj[€dicted 
(the reunion of all predicted segments St+fJ^*^'*"*®*^ and processed holes) and the predicted image 
It+i . According to the invention, the best set of motion parameters Mf,i + A Mt,i is stored with 

a multi-layer motion description. The first layer contains the raw set of motion parameters 
Mli and the flag [YES] or [NO] indicating if the decoder 14 should wait for an additional 

20 layer. The second layer contains the motion parameters correction AM^i • At the end of the 

compression method, the compressed data provided by the encoder 12 are the segmentation 
parameters, the motion parameters Mt,i the best set of motion parameters Mt,i ^ M^i of 

each segment contained in the multi-layer motion description, the overlapping infomiation, 
the holes information and the residual frame Rt+i. 
25 Upon reception of the compressed data, the decoder 14 applies the algorithm 

disclosed on Figure 4. Figure 5 shows the processed frames during the implementation of the 
decompression method. The same algorithm is repeated for each frame Bt+i to be 
decompressed. It is assumed that a previously decoded frame Bt is known and that the 
following frame Bt+i has to be decompressed. 
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At step 402, frame Bt is segmented based on the set of segmentation parameters by 
using the same algorithm and the same settings as its comiter part Bt at the decoder side. 
These settings are set once and for all or are transmitted with encoded frames. The segments 
of Bt are denoted St,i for clarity reason. A segment St,i of Bt is considered at step 404 and the 
5 first layer motion parameters M?,i of segment St,i are decoded and applied at step 406. A 

predicted segment St+i,i is obtained. Steps 404-406 are carried out for all the segments of the 

decompressed frame Bt. 

Overlapping parameters are decoded and applied at step 408 to segment St+i,i ^ 

obtain a new segment denoted St-5,i - Since the reunion of the segments S?H,i may not cover 

1 0 the all frame, -the holes of Bt+i are predicted according to hole information contained in the 
compressed data, at step 410. Then, it is checked in step 412, if the flag contained in the first 
layer of the motion description indicates that additional motion information is contained in 
the second layer. If no additional motion infonnation is contained in the compressed data, 
residual frame decoding is directly performed. A following predicted frame is defined by all 

15 the segments Stili • this point, hole filing is performed, defining a predicted frame Bt+f • 
Residual frame Rt+i is decoded and applied to the predicted frame Bt+f compute the final 

decoded frame Btfi, at step 414. 

If additional motion information is contained in the compressed data, at step 415 

motion parameters correction A M^i is decoded and a corresponding improved segment S^i is 
20 retrieved in the decoded frame Btat step 416. This corresponding improved segment Su is 
the segment of Bt which would be projected on segment St4i,i if the corrected motion 
parameters Mf,i + A M?,i were applied to it. The corrected motion parameters M^i + A Mf,i 
are the raw set of motion parameters Mt,i corrected by the motion parameters correction 
AMli ' At step 417, the projection of Sm using Mli + AM[,i provides a corrected predicted 
25 segment St+t,? • A following con'ected predicted frame is defined by all the segments Sf+u • At 
this point, hole filling is performed, defining a final corrected predicted frame Bfif • Then the 
residual frame Rt+i is decoded and applied to the final corrected predicted fi'ame Bm^ » at step 

418 to provide the final decoded frame Bt+i. 

Advantageously, according to a particular embodiment for a given segment, a 
30 motion vector is defined. This motion vector contains the first and second layers of the 
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motion description. The forward motion parameters are the integer part of this vector. The 
predicted segmentation StVi,i is computed according to this truncated motion information 
contained in the vector by only considering the integer part. 

For the backward motion correction of each predicted segment Sm,\ » 

5 precision vector is used. In this case, there is only one actual layer of motion parameters : 
backward and forward motions are contained in one motion symbol. Still, the motion 
description is two-layered. 

In the previous embodiment, calculation of overlapping parameters at step 210 and 
the corresponding step 408 are performed for a predicted frame Bt+i, after all the segments 

1 0 have been predicted. According to an altemative embodiment, overlapping parameters are 

calculated for each segment after step 208. Thus, they are calculated and encoded segment by 
segment. Accordingly, the overlapping parameters are applied segment by segment just after 
step 406 in the decompression method. 

There are mraierous ways of implementing functions by means of items of hardware 

15 or software, or both. In this respect, the drawings are very diagrammatic, and represent only 
possible embodiments of the invention. Thus, although a drawing shows different fiinctions 
as different blocks, this by no means excludes that a single item of hardware or software 
carries out several fimctions. Nor does it exclude that an assembly of items of hardware or 
software or both carry out a fimction. 

20 The remarks made herein before demonstrate that the detailed description, with 

reference to the drawings, illustrates rather than limits Ihe invention. There are numerous 
alternatives, which fall within the scope of the appended claims. The words "comprising" or 
"comprise" do not exclude the presence of other elements or steps than those listed in a claim. 
The word "a" or "an" preceding an element or step does not exclude the presence of a 

25 pluraUty of such elements or steps. 



